[FLINK-37925][table] Reject out-of-range VARIANT casts and make VARIANT-to-string a value cast by raminqaf · Pull Request #28758 · apache/flink

raminqaf · 2026-07-16T09:55:59Z

What is the purpose of the change

Follow-up to the initial VARIANT-to-primitive cast support (FLINK-37925). It tightens the cast semantics, turns the string cast into a real value cast, adds a decodability check, and documents the resulting behavior.

Numeric casts from a VARIANT now reject values that do not fit the target instead of silently wrapping. An out-of-range integer or an overflowing DECIMAL fails CAST and returns NULL for TRY_CAST. FLOAT and DOUBLE keep lenient IEEE conversion, where overflow becomes infinity. This follows the behavior of Spark's variant casts.
CAST(VARIANT AS CHAR/VARCHAR) now extracts the scalar value, so a stored string is returned unquoted (foo), while objects and arrays return their JSON representation. JSON_STRING remains the way to get the JSON text, where a string stays quoted ("foo"). Previously the cast reused the JSON serialization and was indistinguishable from JSON_STRING.

Brief change log

Add VariantCastUtils (flink-table-runtime): range-checked numeric narrowing (truncate toward zero, throw on overflow) and scalar-to-string extraction.
VariantToPrimitiveCastRule: route integer and DECIMAL targets through the checked helpers; keep FLOAT/DOUBLE lenient.
VariantToStringCastRule: value extraction instead of JSON serialization; JSON_STRING (JsonStringCallGen) is left unchanged.
LogicalTypeCasts: express VARIANT cast validation in the per-target rules and drop the JSON_STRING cast hint.
Fix the Variant.getInstant javadoc to reference Type.TIMESTAMP_LTZ and document microsecond timestamp precision.
Docs: VARIANT value encoding, the PARSE_JSON type mapping, the string-cast vs JSON_STRING distinction, and that NaN/infinity are not valid JSON (witha store-as-string workaround).

Verifying this change

This change added and updated tests:

CastFunctionITCase: overflow cases (CAST fails, TRY_CAST returns NULL), lenient FLOAT/DOUBLE, and string-cast value extraction (a string returns unquoted, objects return JSON).
LogicalTypeCastsTest: the VARIANT cast-support matrix.
BinaryVariantInternalBuilderTest: PARSE_JSON rejects NaN, Infinity, and -Infinity.
BinaryVariantTest: checkFullySupported accepts supported types and rejects every primitive type-code above 16.

Does this pull request potentially affect one of the following parts:

Dependencies (does it add or upgrade a dependency): no
The public API, i.e., is any changed class annotated with @Public(Evolving): yes, javadoc-only change to the @PublicEvolving Variant interface (no signature change)
The serializers: no
The runtime per-record code paths (performance sensitive): yes, only on the VARIANT-to-numeric cast path
Anything that affects deployment or recovery: no
The S3 file system connector: no

Documentation

Does this pull request introduce a new feature? no (it refines existing FLINK-37925 behavior)
If yes, how is the feature documented? docs and JavaDocs

Was generative AI tooling used to co-author this PR?

Yes (please specify the tool below)

Generated-by: Opus 4.8

flinkbot · 2026-07-16T10:01:19Z

CI report:

f9bd3a0 UNKNOWN
ac56197 Azure: SUCCESS

Bot commands

The @flinkbot bot supports the following commands:

@flinkbot run azure re-run the last Azure build

snuyanzin · 2026-07-20T09:27:32Z

+```sql
+CAST(CAST(PARSE_JSON('1000') AS INT) AS TINYINT) -- returns -24 (wrap-around)
+```
+{{< /hint >}}


what is difference here from non VARIANT case?

twalthr

Thank you @raminqaf. I left some feedback.

twalthr · 2026-07-21T13:28:24Z

            LogicalType inputLogicalType,
            LogicalType targetLogicalType) {
-        return methodCall(inputTerm, "toString");
+        return staticCall(VariantCastUtils.class, "toStringValue", inputTerm);


according to LogicalTypeCasts, we support VARCHAR and CHAR both with various length. this cast rule should consider the target length as well. I guess it should throw if length exceeds?

when the target is bounded CHAR(n)/VARCHAR(n), CharVarCharTrimPadCastRule kicks in, calls the variant→string rule, and trims/pads — the identical path every other to-string cast uses.

I added tests to prove it, and they pass:

CAST(PARSE_JSON('"foobar"') AS VARCHAR(3)) → foo (trimmed)

CAST(PARSE_JSON('"ab"') AS CHAR(5)) → ab (padded)

Reverted the change. Now we follow this rule:

VARCHAR(n): OK if length ≤ n, else throw.

CHAR(n): OK only if length == n, else throw.

So CAST (PARSE_JSON('ab') AS VARCHAR(3)) -> returns ab

twalthr

Another round of comments.

twalthr · 2026-07-23T08:14:27Z

+                return staticCall(VariantCastUtils.class, "toLongExact", number);
            case FLOAT:
-                return "floatValue";
+                return methodCall(number, "floatValue");


Are there any data loss checks in this method? Asking AI leads to exactness loss.

can I store a full long in a float?

The short answer is yes, in terms of size (range), but no, in terms of exactness (precision).

If you attempt to store a large long inside a float, the application will not crash, but you will likely lose data through rounding errors.

twalthr · 2026-07-23T08:14:42Z

+                return methodCall(number, "floatValue");
            case DOUBLE:
-                return "doubleValue";
+                return methodCall(number, "doubleValue");


Are there any data loss checks in this method? Asking AI leads to exactness loss.

twalthr · 2026-07-23T08:22:58Z

+
+    public static DecimalData toDecimalExact(Number value, int precision, int scale) {
+        final DecimalData decimal =
+                DecimalData.fromBigDecimal(toBigDecimal(value), precision, scale);


Where is max precision of 38 checked?

Max precision 38 is guaranteed by the target DecimalType itself (DecimalType.MAX_PRECISION), so precision here is always ≤ 38. DecimalData.fromBigDecimal returns null when the value doesn't fit the given precision/scale, and we throw on that. So there's no separate check to add.

… fit the target type and allow string cast Follow-up to the initial VARIANT-to-primitive cast support. Numeric casts from a VARIANT now reject values that do not fit the target type instead of silently wrapping: an out-of-range integer or an overflowing DECIMAL fails CAST and returns NULL for TRY_CAST, matching Spark's variant cast behavior. FLOAT and DOUBLE keep lenient IEEE conversion, where overflow becomes infinity. The new VariantCastUtils performs the checked narrowing by truncating toward zero and range-checking the value. CAST(VARIANT AS CHAR/VARCHAR) is now allowed and returns the JSON string representation, so the previous JSON_STRING-only restriction and its cast hint are removed. VARIANT cast validation is expressed directly in the per-target rules of LogicalTypeCasts. The VARIANT section of the data types reference documents the overflow behavior and the double-cast pattern for wrap-around narrowing.

…pe mapping Describe which value kinds a VARIANT can hold, including that TIMESTAMP and TIMESTAMP_LTZ use microsecond precision and DATE a day count, and that there is no TIME kind. Add a table showing how PARSE_JSON maps JSON values to variant kinds, and explain that PARSE_JSON cannot produce FLOAT, DATE, TIMESTAMP, TIMESTAMP_LTZ, or BYTES because JSON has no literal for them; those kinds come from other producers such as VariantBuilder or format conversions. Point the PARSE_JSON function reference to the VARIANT data type section for the mapping details.

…nt interface Note that getDateTime and getInstant return values with microsecond precision, which the LocalDateTime and Instant return types do not convey on their own. Fix the getInstant javadoc to reference Type.TIMESTAMP_LTZ, the type it actually accepts, instead of Type.TIMESTAMP.

raminqaf force-pushed the FLINK-37925-followup branch 2 times, most recently from beec0e0 to e03672b Compare July 16, 2026 10:07

raminqaf commented Jul 16, 2026

View reviewed changes

Comment thread docs/content/docs/sql/reference/data-types.md Outdated

raminqaf force-pushed the FLINK-37925-followup branch 2 times, most recently from f02dbdc to 49d4bca Compare July 16, 2026 10:45

github-actions Bot added the community-reviewed PR has been reviewed by the community. label Jul 16, 2026

raminqaf force-pushed the FLINK-37925-followup branch 6 times, most recently from 3750b25 to f9bd3a0 Compare July 17, 2026 11:38

raminqaf changed the title ~~[FLINK-37925][table] Range-check VARIANT numeric casts and allow casting VARIANT to string~~ [FLINK-37925][table] Reject out-of-range VARIANT casts and make VARIANT-to-string a value cast Jul 17, 2026

snuyanzin reviewed Jul 17, 2026

View reviewed changes

Comment thread ...k-table-runtime/src/main/java/org/apache/flink/table/runtime/functions/VariantCastUtils.java Outdated

snuyanzin reviewed Jul 17, 2026

View reviewed changes

Comment thread flink-core/src/test/java/org/apache/flink/types/variant/BinaryVariantInternalBuilderTest.java Outdated

snuyanzin reviewed Jul 20, 2026

View reviewed changes

Comment thread ...table-planner/src/test/java/org/apache/flink/table/planner/functions/CastFunctionITCase.java Outdated

davidradl reviewed Jul 20, 2026

View reviewed changes

Comment thread docs/content/docs/sql/reference/data-types.md Outdated

davidradl reviewed Jul 20, 2026

View reviewed changes

Comment thread docs/content/docs/sql/reference/data-types.md

raminqaf force-pushed the FLINK-37925-followup branch from 02f5ffc to e9c07c4 Compare July 21, 2026 09:41

twalthr reviewed Jul 21, 2026

View reviewed changes

raminqaf force-pushed the FLINK-37925-followup branch from 0a0d8d0 to 8f09c44 Compare July 22, 2026 11:00

twalthr reviewed Jul 22, 2026

View reviewed changes

Comment thread ...k-table-runtime/src/main/java/org/apache/flink/table/runtime/functions/VariantCastUtils.java Outdated

raminqaf force-pushed the FLINK-37925-followup branch from 4dc9135 to 23e41c6 Compare July 22, 2026 16:31

twalthr reviewed Jul 23, 2026

View reviewed changes

raminqaf and others added 4 commits July 24, 2026 08:42

[FLINK-37925] Address feedbacks

f096fe4

raminqaf and others added 3 commits July 24, 2026 08:42

[FLINK-37925] Add strikt cast validation for VARCHAR and CHAR

4e2b440

[FLINK-37925] Fail on casting Variant array, objects, bytes to String

91f187c

[FLINK-37925] Address more feedbacks

ac56197

raminqaf force-pushed the FLINK-37925-followup branch from 23e41c6 to ac56197 Compare July 24, 2026 10:55

Uh oh!

Conversation

raminqaf commented Jul 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What is the purpose of the change

Brief change log

Verifying this change

Does this pull request potentially affect one of the following parts:

Documentation

Was generative AI tooling used to co-author this PR?

Uh oh!

flinkbot commented Jul 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

CI report:

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

twalthr left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

twalthr left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

raminqaf commented Jul 16, 2026 •

edited

Loading

flinkbot commented Jul 16, 2026 •

edited

Loading